Papers with pruning techniques
Generalized chart constraints for efficient PCFG and TAG parsing (P18-2)
Copied to clipboard
| Challenge: | Existing pruning techniques limit chart constraints to PCFGs and cannot be applied to more expressive grammars. |
| Approach: | They propose to apply chart constraints to more expressive grammars and a neural tagger which predicts chart constraints at very high precision. |
| Outcome: | The proposed technique accelerates both PCFG and TAG parsing by two orders of magnitude while improving accuracy. |
Enhancing One-Shot Pruned Pre-trained Language Models through Sparse-Dense-Sparse Mechanism (2025.coling-main)
Copied to clipboard
Guanchen Li, Xiandong Zhao, Lian Liu, Zeping Li, Yixing Xu, Dong Li, Lu Tian, Jie He, Ashish Sirasao, Emad Barsoum
| Challenge: | Pre-trained language models (PLMs) are robust in contextual understanding but their considerable size incurs significant computational and storage costs. |
| Approach: | They propose a Sparse-Dense-Sparse pruning framework to prune PLMs . they prune less critical connections using conventional pruning methods . |
| Outcome: | The proposed pruning framework outperforms SparseGPT and Wanda under identical sparsity. |
TAMP: Token-Adaptive Layerwise Pruning in Multimodal Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing pruning methods fail to account for unique token attributes across layers and modalities inherent to MLLMs. |
| Approach: | They propose a pruning framework that takes into account unique token attributes across layers and modalities inherent to MLLMs. |
| Outcome: | The proposed pruning framework outperforms existing pruning techniques on two state-of-the-art MLLMs. |
Successfully Applying the Stabilized Lottery Ticket Hypothesis to the Transformer Architecture (2020.acl-main)
Copied to clipboard
| Challenge: | Current neural networks are heavily growing in depth, with many fully connected layers. |
| Approach: | They propose to combine stabilized lottery ticket pruning with unstructured pruning to improve model performance. |
| Outcome: | The proposed pruning techniques outperform all other techniques for even higher sparsity levels. |
LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluation (2025.coling-main)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is currently the most commonly used PEFT method for fine-tuning models with billions of parameters. |
| Approach: | They propose to use low-rank Adaptation to evaluate LoRA parameter features and then retain LoRA for important layers and the other layers share the same LoRA. |
| Outcome: | The proposed method achieves comparable performance to full fine-tuning and LoRA while retaining 50% of the LoRA parameters on average. |
Data-Informed Global Sparseness in Attention Mechanisms for Deep Neural Networks (2024.lrec-main)
Copied to clipboard
| Challenge: | Attention pruning techniques have been developed to identify and exploit sparseness . previous work has taken pioneering steps to discover and explain the sparsity in attention patterns . |
| Approach: | They propose a framework that observes attention patterns in a fixed dataset and generates a global sparseness mask. |
| Outcome: | The proposed approach saves 90% of computations and maintains quality of results. |
Efficient Unstructured Pruning of Mamba State-Space Models for Resource-Constrained Environments (2025.emnlp-main)
Copied to clipboard
| Challenge: | State-space models struggle with quadratic computational complexity, limiting their use in long-context tasks and resource-constrained input data. |
| Approach: | They propose a pruning framework specifically tailored for Mamba that reduces parameter counts by 70% with only a 3–9% drop in performance. |
| Outcome: | The proposed pruning framework achieves up to 70% parameter reduction with only a 3–9% drop in performance. |
Pruning Foundation Models for High Accuracy without Retraining (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Despite the superior performance of foundation models, it is challenging to deploy large language models in practical applications due to their massive parameters and computations. |
| Approach: | They propose a pruning algorithm to prune LLMs in one-shot without retraining . they propose retrainable pruning algorithms to prune multiple weights in LLM . |
| Outcome: | The proposed pruning methods perform better than baseline pruning methods on sparse and unstructured sparsity models. |
Word-Level Coreference Resolution (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent coreference resolution models rely heavily on span representations to find coreference links between word spans. |
| Approach: | They propose to consider coreference links between individual words rather than word spans and reconstruct the word span. |
| Outcome: | The proposed model outperforms existing models on the OntoNotes benchmark while being highly efficient. |
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect (2025.findings-acl)
Copied to clipboard
Xin Men, Mingyu Xu, Qingyu Zhang, Qianhao Yuan, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, Weipeng Chen
| Challenge: | Recent studies have identified significant redundancy in large language models . quantization and pruning are two methods that reduce computational resources . |
| Approach: | They propose simple pruning methods that prune redundant layers based on their BI scores. |
| Outcome: | The proposed pruning methods demonstrate superior performance over previous pruning methods. |
GAP: a Global Adaptive Pruning Method for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing structured pruning methods employ uniform compression rates across network layers, neglecting the varying importance of different network depths. |
| Approach: | They propose a pruning framework that minimizes global capability loss by layer-adaptive pruning rates. |
| Outcome: | The proposed approach achieves comparable performance with state-of-the-art methods at high pruning rates and shows significant advantages at low pruning rates. |
xCOMET-lite: Bridging the Gap Between Efficiency and Quality in Learned MT Evaluation Metrics (2024.emnlp-main)
Copied to clipboard
| Challenge: | State-of-the-art trainable machine translation evaluation metrics rely on large encoders . this makes them computationally expensive and inaccessible to researchers with limited resources. |
| Approach: | They propose a method to extract knowledge stored in large encoders and a pipeline for efficient black-box distillation. |
| Outcome: | The proposed model surpasses COMET-22 and BLEURT-20 on the WMT22 dataset by 6.4%. |